HW-assisted upscaling and multi-sampling using a high resolution depth buffer

ABSTRACT

One aspect of the current disclosure provides a method of upscaling an image. The method includes: rendering an image, wherein the rendering includes generating color samples of the image at a first resolution and depth samples of the image at a second resolution, which is higher than the first resolution; and upscaling the image to an upscaled image at a third resolution, which is higher than the first resolution, using the color samples and the depth samples.

CROSS REFERENCE TO RELATED APPLICATION

The present application is also related to U.S. patent application Ser.No. 15/967,645, filed May 1, 2018, entitled “ADAPTIVE UPSCALING OF CLOUDRENDERED GRAPHICS” naming Rouslan Dimitrov, et. al, as inventors, andincorporated herein by reference in its entirety.

TECHNICAL FIELD

This application is directed, in general, to a video upscaling andrendering process and, more specifically, to managing resources of aserver system through the video rendering process.

BACKGROUND

In a cloud based application service, images can be rendered as part ofan executing application, encoded into a video stream, and thendelivered to a client computing device for display to a user. Certainapplications can be more resource intensive than others. For example, afast moving game application can require significantly more graphicalprocessing to render images than a CAD application. On a physical serversystem or sets of servers, the amount of resources, for video renderingis limited, i.e., there is a maximum level of computational resourcesavailable to be used by applications.

As additional applications are started and actively running on a serversystem, such as when additional application virtual machines (VMs) arecreated, the available resources of the server system are allocated toeach of the VMs. Due to the physical limitation of the maximumcomputational resources available, there will be a maximum number of VMsthat can actively be run on the server system before there is adegradation in response and execution speed of the application VM.Currently, the industry method of resolving this issue is to addadditional servers systems to support additional application VMs,thereby adding cost to the environment for the hardware, the physicalspace required, and the system engineers to maintain the additionalserver systems.

To display an image with the optimal display quality, especially on aliquid crystal (LCD), organic light-emitting diode (OLED), or other kindof flat panel display, one has to render the image at a resolution thatmatches the native resolution of the display. As the native resolutionscontinue to increase, from the current ˜8 million pixels (4K) to ˜32million pixels (8K) or even more, rendering an image at these highresolutions can be very expensive in terms of resource and time. Also,when an image is rendered remotely and has to be transmitted to thedisplay, the higher resolutions place an increasing burden on thecommunications link. It is noted that these displays can span a widerange of physical dimensions, from the few inches of a smartphonescreen, to many feet for a large flat panel.

SUMMARY

One aspect of the current disclosure provides a method of upscaling animage. The method includes: rendering an image, wherein the renderingincludes generating color samples of the image at a first resolution anddepth samples of the image at a second resolution, which is higher thanthe first resolution; and upscaling the image to an upscaled image at athird resolution, which is higher than the first resolution, using thecolor samples and the depth samples.

Another aspect of the current disclosure provides a system for upscalingan image. The system includes: a processor includes: a rendering engineconfigured to render an image by generating color samples of the imageat a first resolution and depth samples of the image at a secondresolution, which is higher than the first resolution; and an upscalingengine configured to upscale the image to an upscaled image at a thirdresolution, which is higher than the first resolution, using the colorsamples and the depth samples; and a memory coupled to the processor andincludes a color buffer storing the color samples and a depth bufferstoring the depth samples.

Yet another aspect of the current disclosure provides a computer programproduct. The computer program product, when executed by one or moreprocessors, causes the one or more processors to: render an image bygenerating color samples of the image at a first resolution and depthsamples of the image at a second resolution, which is higher than thefirst resolution; and upscale the image to an upscaled image at thesecond resolution using the color samples and the depth samples.

BRIEF DESCRIPTION

Reference is now made to the following descriptions taken in conjunctionwith the accompanying drawings, in which:

FIG. 1 illustrates a block diagram of an embodiment of a video renderingsystem constructed according to the principles of the disclosure;

FIG. 2 illustrates a block diagram of an example video/image renderingsystem that is constructed according to the principles of currentdisclosure;

FIG. 3 illustrates a block diagram of an example processor for upscalingan image implemented according to the principles of current disclosure;

FIG. 4 illustrates a flow diagram of an example method for upscaling animage carried out according to the principles of current disclosure;

FIG. 5A illustrates a flow diagram of an example method for upscaling animage carried out a according to the principles of current disclosure;

FIG. 5B illustrates an example of a grid with 4:1 mapping between colorand depth samples of an image;

FIG. 5C illustrates another view of the grid of FIG. 5B with slopesbetween depth samples;

FIG. 5D illustrates an exemplary pair of with the slopes in the grid ofFIGS. 5B and 5C;

FIG. 5E illustrates yet another view of the grid of FIGS. 5B and 5C;

FIG. 5F illustrates an example of a grid with interior depth samples;

FIG. 5G illustrates an example of a grid that has been logicallyrotated;

FIG. 5H illustrates another example of a grid that has been logicallyrotated;

FIG. 5I another view of the grid of FIG. 5F with seven 45° pairs of themapped color samples;

FIG. 5J illustrates another view of the grid of FIG. 5F with seven 135°pairs of the mapped color samples;

FIG. 5K illustrates an exemplary graph showing dominating directions;

FIG. 5L illustrates another view of the grid of FIG. 5F with adominating direction of 45°;

FIG. 5M illustrates another view of the grid of FIG. 5F with adomination direction of 135°;

FIG. 5N illustrates another view of the grid of FIG. 5F with nodominating direction; and

FIG. 5O illustrates a grid with 2:1 mapping between color and depthsamples of an image;

FIG. 6 illustrates a flow diagram of an example method for managingserver system resources with respect to rendering images for a video;

FIG. 7 illustrates a flow diagram of an example method for the usage ofvarious compression bit stream algorithms;

FIG. 8 illustrates a block diagram of an example video rendering systemcapable of delivering a video to a client computing device;

FIG. 9 illustrates a block diagram of an example compression algorithmsystem flow; and

FIG. 10 illustrates different connection cases and the additionalinformation of sharpening when determined on the server that can be usedfor a filter map.

DETAILED DESCRIPTION

Rendering can be a resource intensive process that uses multiple passesto convert data into a format capable of being displayed on a displaydevice. The rendering process can take into account various embodiments,variations, and scene parameters, such as determining if one objectblocks another and to account for lighting and shading of a scene, tocompute the color values for the pixels that will be displayed in eachframe. The rendering process can be initially executed on a serversystem or set of server systems (server systems), such as a cloud-basedrenderer or a data center environment.

Real-time rendering can be used to render images or frames that are thenencoded to form a video that can be delivered to client computingdevices for display to one or more users. The video utilizes bandwidthand the amount of data in the video directly drives the time it takes tocomplete the transmission of the video to the client computing device.

Reducing the amount of computational and other resources associated withserver systems that are needed to render images can be beneficial to theindustry. Methods to shift some or all of the rendering to lower costsystems, i.e. adaptive rendering, such as client computing devices,allows a greater number of virtual machines (VMs) to exist on the serversystems. This can increase the VM density, i.e., the number of VMspresent in a single physical host which can be run normally without theVMs being starved for a particular resource. Rendering color pixels at alower resolution and then employing upscaling at the server system, suchas discussed above with respect to FIG. 2 to FIG. 5, can also be used toreduce computational and other resource requirements at server systems.

As additional VMs are created on a server, such as when additionalclient computing devices connect to the environment, the availableresources are reduced. When a certain number of VMs are actively runningon the physical servers, it is possible that all available resources arebeing utilized. This number of VMs can vary due to several factors, forexample, the hardware being used for the physical server, and theresources required for each of the running VMs. Increasing the number ofVMs past the determined maximum number of VMs, i.e. oversubscription,can have the effect that computational resources need to be shared,thereby causing slower response times for processing requests andunpredictable instantaneous resource availability. For example, a videodelivered at 60.0 frames per second (FPS) can drop to 30.0 FPS due tothe restricted availability of the computational resources, such ascentral processing units (CPUs) and graphical processing units (GPUs).

To increase efficiency on the server system, video rendering processescan be divided so that on the server system VM, rendering can beperformed at a lower resolution and then the upscaling is performed on aclient computing device. Rendering data, which can include a lowresolution color buffer and connectivity information, can be sent to theclient computing device for final rendering (e.g., upscaling) fordisplay on the client computing device. The rendering data can becompressed using an available algorithm and then sent to the clientcomputing device where the client computing device uses the renderingdata to generate video for the client display device. Upscalingprocesses can be executed on the client computing device therebyreducing the resources, i.e. GPUs, processors, memory, and bandwidth,utilized within the server system and application VM. The applicationthat is requesting the video to be displayed can have minimal or nomodification in order to take advantage of the adaptive renderingtechniques described herein. In the case of implicit upscaling, theprocess can use heuristics, e.g., an application rendering state andapplication hints, to detect the screen space depth buffer and instructa GPU, on behalf of the application, to render a low resolution colorbuffer and a high resolution screen space depth buffer, i.e.depth-stencil buffer. The reduction in computational resources utilizedby the VMs allow additional VMs to be actively running on the serversystem, thereby increasing the VM density and lowering systemoperational costs.

The algorithm and heuristics used and the parameters utilized ingenerating and compressing the rendering data, can be based on andmodified by several factors. For example, the available bandwidthbetween the application VM and the client computing device can determinethe level of lossy compression that may be necessary to maintain aminimum target FPS. The application VM can also request information fromthe client computing device regarding the capabilities of the clientvideo component, such as built-in optimization protocols, memory,processing power, and other factors. An additional parameter that can beconsidered by the application VM is the capability of the client displaydevice. For example, the client display device's resolution capabilitycan affect the amount of rendering data needed to be sent to the clientcomputing device to maintain an adequate visual experience for the userof the client computing device.

The algorithm used in pre-processing rendering data to be sent to aclient computing device can utilize different factors. For example, thedepth pixel count can vary as to the color pixel count. Ratios ofvarious combinations can be used, with the ratio of four depth pixelsfor each color pixel being described herein. Various ratios, typicallybased on a power of two, can be utilized, such as two, eight, orsixteen, depth pixels for each color pixel, or other such combinations.Determining an appropriate depth/color ratio can result in thecapability to deliver a desired framerate or video quality, for example,3,840 pixels across 2,160 lines (2160p or 4K) resolution.

A pixel connectivity map can be obtained from the depth buffer and thencompressed to a filter map, which indicates to the client computingdevice how the color buffer should be filtered. Typically, a pixelconnectivity map contains eight bits per depth value, indicating theconnectivity in the eight possible directions (shown in FIG. 5C,starting with pixel 570-X and examining the path to the eight pixels570-C(x)). The filter map takes as input the pixel connectivity map fora two by two (2×2) non-overlapping tile of pixels and compresses thedata into fewer bits for each 2×2 tile. The filter map can be generatedby determining the number of connected pixels of the target pixel. Ifthere are 0 or 4 pixels connected, then an interpolator is used for thepixel, using a 1×4 or 4×4 filter kernel. If there is 1 pixel connected,then the value of the connected pixel is used as the value of the targetpixel. If there are 2 pixels connected, the mean of the 2 connectedpixels are used as the value of the target pixel. If there are 3 pixelsconnected, the mean of the 2 pixels on the long diagonal is used as thevalue for the target pixel.

A block can be of various sizes, such as 8×8 pixels or 16×16 pixels.Once the filter map and color buffer are determined, they each can besent to the client computing device as rendering data for rendering andupscaling to appropriate client display resolution. Compressiontechniques can be applied to the rendering data when sending to theclient computing device. See FIG. 7 and its accompanying description fora description of some of the compression bit stream algorithms that canbe applied.

The delay between when the frame is being rendered, i.e. a determinationof what will be displayed and when a video is rendered for display on aclient computing device, can be driven by several factors. These factorscan also drive the decision process for determining where the video willbe rendered, i.e. on the server systems' video processor or on theclient computing device. For example, some of the factors and parametersare: the availability of VM or resources at the server, the bandwidthavailable between the server and the client computing device, theconnectivity data throughput between the server and the client computingdevice, the client payment tier level, and a minimum target FPS for thevideo for the application that is executing. Depending on theapplication being executed, a minimum target FPS can be set at 30.0,60.0, or another appropriate value. For example, a CAD application videocan have a minimum target FPS of 30.0 FPS while a game application canhave a minimum target FPS set at 60.0 FPS.

In another embodiment, the user of the client computing device canpurchase, through various means, a higher tier level of service, forexample, the application system should maintain a higher FPS rather thandropping to a lower FPS under conditions where the application serverresources become heavily allocated. In addition, the user can purchase atier of service that provides for a higher resolution video to bedelivered to the client computing device. In this circumstance, theapplication VM can utilize certain algorithms to provide the renderingdata in a state to allow upscaling to the desired resolution. In anotherembodiment, the client computing device, utilizing applications andtools downloaded from the server system, can provide for video imageenhancement, such as removing compression artifacts, spatial andtemporal anti-aliasing, and modifying the algorithm parameters.

In this disclosure, a video is a stream of rendered images and includessome or all of a portion, segment, frame, scene, or snippet of videothat is displayed. For example, the video can be a rendered video scenefor a CAD software application or an active game play scene in a gameapplication. Rendering a video refers to rendering images that are thenencoded to form a video stream. A client computing device can be acomputing device or system coupled directly to the application system,such as through a wired or wireless network connection, or it can becommunicatively coupled such as through a type of internet connection.

Turning now to the Figures, FIG. 1 illustrates a block diagram of anembodiment of a video rendering system 100 constructed according to theprinciples of the disclosure. The video rendering system 100 includes aserver system 110, client computing devices 120, 130, 140, 150, 160, anda communications network 170. The client computing devices 120, 130,140, 150, 160, are collectively referred to as client computing devices120-160, and are communicatively coupled to the server system 110 viathe communications network 170. The communications network 170 can be aconventional network, such as the internet, private network, or othertype of network, that allows connected client computing devices 120-160to communicate with the server system 110 and an option to communicatewith each other.

The server system 110 is configured to render images and generate videofrom the rendered images to send to the client computing devices120-160. The server system 110 can be implemented on a single server oron multiple servers of a cloud computing platform, data center, or otherserver environment. The server system 110 can include at least one CPUand multiple GPUs. VM's can be created where the CPU and GPU's areallocated to the VMs to provide server-based rendering. In theillustrated embodiment, the server system 110 includes a renderer 111, avideo encoder 112, a video transmitter 113, a memory 114, and anapplication engine 115. The memory 114 can be a conventional memory ormemories typically employed with servers. The application engine 115includes the operating instructions that correspond to the algorithmsemployed to generate scenes, such as a game engine providing scenes froma video game.

The renderer 111 generates a set of images that are encoded into a videofor transmission to the client computing devices 120-160 via the videotransmitter 113 and the communications network 170. The renderer 111 canbe a cloud-based and server-based renderer. The rendered imagescorrespond to application data received from, for example, the clientcomputing devices 120-160 and the application engine 115. Theapplication data can include scene data.

The renderer 111 can include various computing resources including bothCPUs and GPUs. For example, Nvidia Grid™ technology can be employed toprovide the renderer and rendering schemes disclosed herein to supportrendering of the images, such as disclosed in FIGS. 2 to 5 herein.Nvidia Grid™ is by Nvidia Corporation of Santa Clara, Calif., andprovides a graphics virtualization platform that allows the power ofNvidia GPU technology to be used by virtual environments. The renderer111 can include additional components to those illustrated in FIG. 1such as typically included in a cloud and server based renderer.

The video encoder 112 encodes the rendered images into a video fortransmission. The video encoder 112 can also provide additionalfunctions such as reformatting and image processing. The encodedrendered images are then provided to the video transmitter 113 and sentto the client computing devices 120-160. The video transmitter 113 canbe a conventional device that receives encoded frames and transmits themas a video stream. Instead of being included within the server system110, in some embodiments the video transmitter 113 can be conventionallycoupled to the video rendering system 100. In some embodiments, thevideo transmitter 113 is a video proxy server. As disclosed herein, insome embodiments rendering data can be sent by the video transmitter 113to use for upscaling at the client computing devices 120-160.

The client computing devices 120-160 can be VR headgear, smartphones,desk top computers, laptops, computing pads, tablets, etc. The clientcomputing devices 120-160 can be thin clients that communicate with theserver system 110 and provide sufficient application data thereto forrendering. Each of or at least some of the client computing devices120-160 can be different types of devices. For example, client computingdevices 120-140 can be VR headgears, computing device 150 can be alaptop, and computing device 160 can be an Nvidia SHIELD Tablet.

FIG. 2 illustrates a block diagram of an embodiment of a video/imagerendering system 200 constructed according to the principles of thedisclosure. In the illustrated embodiment, the video rendering system200 includes a server system 210 and a client computing device 250. Theserver system 210 includes a processor 212, a memory 214, and a videoencoder 218, all of which are connected to one another via aconventional means. The processor 212 may include one or more CPUs (notshown) and one or more co-processors, which may take the form of one ormore GPUs (not shown). The memory 214 may be any one or more ofread-only memory (ROM), volatile random-access memory (RAM) andsolid-state or hard drive storage unit. It is understood that althoughnot illustrated, the server system 210 may also include otherconventional components of a server.

The client computing device 250 includes a client display 252 and avideo decoder 254. The client computing device 250 can becommunicatively coupled to server system 210 by various availablecommunication types, such as an internet connection and a type ofnetwork connection, such as an Ethernet or wireless connection. Althoughnot illustrated, the client computing device 250 may include otherconventional components of a computing device, such as a processor and amemory.

In the illustrated embodiment, the processor 212 renders an image andupscales the rendered image using the color and depth samples thereof.The processor 212 determines connection information from the depthsamples of the rendered image and uses the connection information withthe color samples of the rendered image to upscale the rendered imageinto a high-resolution image, such as an image at the second resolution.The processor 212 provides the upscaled image to the video encoder 218for encoding and transmission to the client computing device 250.

In the illustrated embodiment, the memory 214 includes a low-resolutioncolor buffer 215 that stores the color samples (and values thereof) ofthe rendered image and a high-resolution depth buffer 216 that storesthe depth samples (and values thereof) of the rendered image. The colorsamples are sampled at a first resolution and the depth samples aresampled at a second resolution that is higher than the first resolution.In one embodiment, the first resolution is 1,920 pixels across 1,080lines (1080p) and the second resolution is 4K. In another embodiment,the first resolution is 4K and the second resolution is 7,680 pixelsacross 4,320 lines (4320p or 8K).

In one embodiment, before providing the upscaled image to the videoencoder 218, the processor 212 can anti-alias the upscaled image. Onemethod for anti-aliasing can be to apply an algorithm such as fastapproximate anti-aliasing (FXAA) or temporal anti-aliasing (TXAA) to theupscaled image. A second method that can be used for anti-aliasing canbe to apply a filter to the upscaled image and downscale it back down tothe lower resolution. The filter can give the new low resolution image ahigher quality than the original image. It is understood thatanti-aliasing is an optional process and its applicability is based onthe resolutions of the buffers (and samples therein) and the targetresolution. The resolution of the anti-aliased image can be the same asthe first resolution, which is the resolution of the rendered image'scolor samples. It is also understood that, based on the targetresolution of the anti-aliased image, the resolutions of the buffers andthe samples therein may be adjusted.

In the illustrated embodiment, the video encoder 218 encodes the image(and color samples of thereof) received from the processor and transmitsthe encoded images as a video stream, or video, over the network to theclient computing device 250. Although not illustrated, the video encoder218 can include a network interface that is configured to transmit theencoded images to the video decoder 254. In some embodiments, the serversystem 210 can include a video transmitter, such as the videotransmitter 113 of FIG. 1, which transmits the video of encoded imagesto the client computing device 250.

In the illustrated embodiment, the video decoder 254 receives anddecodes the encoded images from the server system 210. Although notillustrated, the video decoder 254 can include a network interface thatis configured to receive the encoded images from the video encoder 218.Once decoded, the images from the server system 210 are provided to thedisplay 252. In the illustrated embodiment, the upscaled images aredisplayed by the display 252 at a native resolution of the display 252.In one embodiment where the upscaled images are sent withoutanti-aliasing, the native resolution may be same as the secondresolution, i.e. the resolution of the high-resolution depth buffer. Inanother embodiment where the anti-aliasing is performed on the upscaledimages, the native resolution may be the same as the first resolution,i.e. the resolution of the low-resolution depth buffer. It is understoodthat, based on the intended resolution of the displayed image, the firstand second resolutions of the samples may be adjusted.

FIG. 3 illustrates a block diagram of an example processor 300 forupscaling an image implemented according to the principles of thecurrent disclosure. The processor 300 corresponds to a processor of aserver system in a client-server environment such as shown in FIG. 2. Inthe illustrated embodiment, the processor 300 includes various graphicsprocessing engines including a rendering engine 310, an upscaling engine320, and an anti-aliasing engine 330. It is understood that each ofthese graphics processing engines, 310, 320 and 330 may correspond toalgorithms represented by a different portion, e.g., a set ofinstructions, of a computer program product that, when executed by oneor more processor, causes the executing processor to perform certainfunctions.

In the illustrated embodiment, the rendering engine 310 renders an imageby generating color samples of the image at a first resolution. Therendering engine 310 also generates depth samples of the image at asecond resolution that are higher than the first resolution of the colorsamples. The color and depth samples are generated from a singlerendering pass. Color and depth buffers storing these samples may belocated in a memory coupled to the processor 300, for example, a memorysuch as the memory 214 in FIG. 2.

In the illustrated embodiment, the upscaling engine 320 upscales therendered image to an upscaled image at a third resolution that is higherthan the first resolution. The third resolution may be as high as thesecond resolution, i.e. the resolution of the depth samples. Theupscaling engine 320 interpolates missing color values for the thirdresolution from the generated color and depth samples.

In the illustrated embodiment, the anti-aliasing (AA) engine 330anti-aliases the upscaled image. To avoid the cost, e.g., the memoryfootprint and the bandwidth, of reading/writing the high-resolutionupscaled image to an intermediate color buffer, the color samples of theupscaled image are fed directly into the AA engine 330. The upscaling ofthe upscaling engine 320 and the anti-aliasing of the AA engine 330 maybe performed in a single pass.

The AA engine 330 anti-aliases all geometry edges in the upscaled imageby resolving the upscaled image back to a lower resolution. The AAengine 330 can divide color samples of the upscaled image into groups,e.g. groups of four adjacent color samples, and take an average of eachgroup. For an easier and simpler implementation of anti-aliasing, thegrouping/down sampling ratio may be same as the mapping ratio used inthe upscaling. In such a case, the anti-aliased image would have thesame resolution as the rendered image, e.g., the first resolution. In anembodiment where the resolution of the upscaled color samples is thesame as the target resolution, e.g. display/native resolution,techniques such as fast approximate anti-aliasing (FXAA) or temporalanti-aliasing (TXAA) may be applied to the upscaled color samples. Inanother embodiment, the AA engine 330 may use a nonlinear filter or ahigh-order anti-aliasing filter such as Catmull-Rom filter, a Gaussianfilter or Mitchell-Netravali filter for a high-quality anti-aliasing. Insuch an embodiment, each output pixel is computed as a weighted averageof a 4×4 neighborhood of samples in the upscaled image.

It is understood that the anti-aliasing of the AA engine 310 is anoptional step and may be omitted. For example, in an embodiment wherethe required resolution, e.g., the native resolution of a displaydevice, is 4K and the upscaled image is already at such a resolution,the anti-aliasing of the AA engine 330 may be omitted to keep the imageat the 4K resolution. In an embodiment where the required resolution isless than the upscaled image, the anti-aliasing of the AA engine 330 maybe performed to produce a higher quality image having mitigated aliasingartifacts. FIG. 3 is a demonstration of the functionality and featuresbeing described herein and each of the components described can bephysically combined or separated into one or more processors, of varioustypes, as appropriate.

FIG. 4 illustrates a flow diagram of an embodiment of a method forupscaling an image. The method 400 may be performed by a server system,such as the server system 200 in FIG. 2. The method 400 begins at step405.

At step 410, an image is rendered. The step 410 may be performed by arendering engine executed by a processor such as the rendering engine310 in FIG. 3. Rendering the image includes generating color and depthsamples of the image and storing those samples in respective buffers. Inthe illustrated embodiment, the color samples of the image, which aregenerated at a first resolution, are stored in a color buffer such asthe low-resolution color buffer 215 in FIG. 2, and the depth samples ofthe image, which are generated at a second resolution, are stored in adepth buffer of a memory such as the high-resolution buffer 216 in FIG.2.

In one embodiment, the number of color samples, i.e., the firstresolution, is one-fourth the number of depth samples, i.e., the secondresolution, and the color samples would be generated once in a singlepass with the depth samples. Each generated color sample wouldcorrespond to 2×2 depth samples in a grid. This gives a consistent 4:1(or “4×”) ratio between the number of depth and color samples. Thismapping is shown in more detail in FIG. 5B.

In another embodiment, the number of color samples, i.e., the firstresolution, is one-half the number of depth samples, i.e., the secondresolution, giving a 2:1 (or “2×”) ratio between the number of depth andcolor samples. This can be achieved by generating the color samplestwice at the 4× ratio of the previous embodiment, but at differentoffsets. In more detail, for a consistent mapping between the color anddepth samples, a first set of the color samples would be generated firstfor the Original position and a second set of the color samples would begenerated for the Diagonal position. The term “Original position” refersto one quadrant in each sample quad, to which each color sample of thefirst set maps, and the term Diagonal position refers to anotherquadrant in each sample quad situated diagonally to/from the Originalposition, to which each color sample of the second set maps. It isunderstood that the order of generating color samples and the positionof the generated color samples may change as long as the two samediagonally positioned quadrants in each sample quad are mappedrespectively by the color samples from the first and second sets. Thismapping is shown and discussed further with FIG. 5O.

In some embodiments, the position mapping between color samples anddepth samples may be dynamically determined at rendering time. In suchcases, the color samples are not affixed to a predetermined set of depthsamples. Instead, each color sample is considered as shared by all depthsample locations under the (low-resolution) pixel region that it coversduring rendering. The shader and hardware are programmed to dynamicallyselect which fragment generating a depth sample can update a colorsample. This selection process is necessary when samples generated by afragment passing depth test only partially cover the depth sampleswithin a color pixel region. There are several selection strategies:

-   -   SAMPLEID: the fragment with the pre-specified depth sample ID        writes to color;    -   LAST: the last fragment arrives (in API order) writes to color;    -   FIRST: the first fragment that survives the z-test writes to        color;    -   TOP: the fragment that has the smallest (post z-test) depth        value writes to color;    -   BOTTOM: the fragment that has the largest (post z-test) depth        writes to color;    -   COVMAX: the last primitive that has the largest coverage mask        (in terms of number of samples it covers in the pixel) writes to        color;    -   COVMIN: the last primitive that has the smallest coverage mask        writes to color;    -   PRIMMIN: the smallest or skinniest primitive (therefore less        likely be covered or kept in neighbor) writes to color.

Among the listed strategies, SAMPLEID and LAST are directly available onexisting hardware. FIRST and BOTTOM are available but require two passesof scene rasterization. The rest can be realized either infixed-function hardware or by a software implementation withpixel-shader interlock feature. Some of these strategies aim to maximizethe likelihood of getting each depth pixel a correct anchor color in itsneighborhood. This is usually preferred in cases where thin geometry(either foreground or background) exists. It is understood that thesestrategies can be combined and employed in a checkerboard or similarpattern, e.g. odd pixels using TOP and even pixels using BOTTOM. For thecase where more than one color sample are stored at each pixel, thesecolor samples can use different strategies to maximize the likelihoodthat both foreground and background color are stored at eachlow-resolution pixel region. The pattern can also alternate the selectedstrategies in every frame, thus providing temporal variations to furtherreduce perception of pixel error.

It is understood that unlike conventional upscaling methods, theresolution of the generated depth samples are higher than the colorsamples and are not discarded after the image is rendered. It is alsounderstood that the image is rendered with textures using an additionallevel of detail bias (LOD BIAS) of −1, added to any LOD BIAS alreadypresent.

Referring back to FIG. 4, the image rendered at the step 410 is upscaledto a third resolution, which is higher than the first resolution, atstep 420. The step 420 may be performed by an upscaling engine executedby a processor, such as the upscaling engine 320 in FIG. 3. The thirdresolution may be as high as the second resolution, i.e., the resolutionof the depth samples. In the illustrated embodiment, the rendered imageis upscaled using the generated color samples and the connectioninformation between the generated depth samples of the rendered image.The connection information may be in the form of a connectivity map.Details on how the connection information can be determined and used forupscaling are discussed in FIG. 5A to FIG. 5O.

At step 430, the upscaled image is displayed on a display device. Thedisplay device can be, for example, a smartphone or another displaydevice having a high dots/pixels per inch, such as 300 or more dots perinch. Parameters of the upscaling, e.g., the resolutions and mapping ofthe generated samples, may be adjusted to meet the requirement, e.g. thenative resolution, of the high-DPI display for optimal viewing. Themethod 400 ends at step 435.

In an alternative embodiment, the upscaled image may be anti-aliased atstep 425 before the step 430. This embodiment would be ideal for asituation where the requirement, e.g., the required resolution of thefinal image, is less than the upscaled image, since anti-aliasing lowersthe resolution of an image. The step 425 may be performed by ananti-aliasing engine executed by a processor, such as the AA engine 330in FIG. 3

In this embodiment, the color samples of the upscaled image from thestep 420 are fed directly into the anti-aliasing engine. As mentionedabove, this saves the memory footprint and the bandwidth that would bespent on writing and reading the color values of the upscaled image froman intermediate color buffer. The color samples of the upscaled image isthen anti-aliased to a lower resolution, such as the first resolution,to remove artifacts, e.g. remove jagged geometry edges. The upscalingstep 420 and the anti-aliasing step 425 may be performed in a same pass.The anti-aliased image is then displayed on a display device at the step430. The method 400 ends at the step 435.

In general terms, the upscaling step 420 computes color value for eachdepth sample at high resolution, e.g., the second resolution, based oncolor samples at low resolution, e.g., the first resolution, and theconnection between the depth sample and its surrounding color samples.This connection can be determined by checking the consistency of depthbetween the color samples and the depth sample. The consistency of depthis based on a Cl (slope) continuity check of the depth values. In oneembodiment, depth slopes can be computed by finite differencing ofadjacent depth samples, such as the method 500 detailed below. Inanother embodiment, depth slopes can be computed in fragment shaders byusing quad derivative intrinsic functions, and are stored alongside withcolor samples.

FIG. 5A illustrates a flow diagram of an embodiment of a method 500 forupscaling an image carried out according to the principles of thecurrent disclosure. The method 500 corresponds to the step 420 in FIG.4. The method 500 is described below with respect to 4X upscaling, whichimproves a resolution of an input (originally rendered) image by four(4) times. The method 500 uses the rendered image's color samples thatare generated at a first resolution and the rendered image's depthsamples that are generated at a second resolution. The method 500 can beperformed by an upscaling engine executed by a processor, such as theupscaling engine 320 in FIG. 3. The method starts at a step 505, whichis after an image has been rendered and the image's color and depthsamples have been generated and stored in respective buffers.

At step 510, color samples of the rendered image are mapped tocorresponding depth samples. As the second resolution is four timeshigher than the first resolution, there are four times more depthsamples than the color samples and 4:1 (depth samples to color samples)mapping is performed. FIG. 5B illustrates this 4:1 mapping, where eachcolor sample maps to every fourth depth sample. Looking at the grid 560in FIG. 5B as a collection of quads, i.e. 2 samples×2 samples, eachcolor sample maps to a depth sample in the same quadrant in each quad.Each circle denotes a color sample, each square denotes a depth sample,and each circled-square denotes a mapped color-depth pair. For clarityin FIG. 5B, only two of the circles are representatively identified as562 and only two of the squares are representatively identified as 570.

At step 520, for each depth sample, their connection(s) with contiguousdepth samples are determined. It is understood that the term “contiguousdepth sample” refers to a depth sample that directly abuts a given depthsample.

To determine whether each depth sample is connected to its contiguousdepth samples, the step 520 computes first slopes, i.e., dashed slopes,between each depth sample, e.g., 570-X, and the contiguous depthsamples, e.g., 570-C's in FIG. 5C, and second slopes, i.e., solidslopes, between the contiguous depth samples, e.g., 570-C's, and thedirectionally adjacent depth samples, e.g., 570-A's. An exemplary pairof slopes is shown in FIG. 5D. It is understood that the term“directionally adjacent depth sample” refer to a depth sample thatdirectly abuts the contiguous depth sample in a same direction as thecontiguous depth sample abuts the unmapped depth sample.

The step 520 then calculates a difference between each respectivefirst-second slope pair and compares the absolute value of thedifference to a threshold. Based on this comparison, the connectionsbetween each depth sample and its contiguous depth samples aredetermined. This can be described as:

$\begin{matrix}{{{{\frac{z_{origin} - z_{contiguous}}{\Delta\; d} - \frac{z_{contiguous} - z_{directional}}{\Delta\; d}}} < t},} & (1)\end{matrix}$where z_(origin) corresponds to the depth value 570-X in FIG. 5D,z_(contiguous) to 570-C, and z_(directional) to 570-A. Ad corresponds tothe distance between the depth values, and t is the threshold value.Note that t is one of the algorithm parameters that can be modified forproviding video image enhancement.

Since Δd is positive, the equation (1) can be rewritten as |z_(origin)−2z_(contiguous)+z_(directional)|<t Δd (2). It is noted that Δd is notnecessarily 1, and that it is possible for a given depth sample to beconnected to the contiguous depth sample, but the contiguous depthsample not connected to the given depth sample. For depth sampleslocated in a diagonal slope, such as 570-X, 570-Cd and 570-Δd in FIG.5C, the threshold may be multiplied by a factor of √{square root over(2)} though this is not required.

In one embodiment, instead of comparing the differences of the slopes,the step 520 may use a threshold to measure how far the depth value ofeach depth sample is from the other depth samples in a given direction.In such an embodiment, the distance (in terms of depth value) between agiven depth sample, e.g. 570-X, and a contiguous depth sample, e.g.,570-Cd and a directionally adjacent depth sample, e.g., 570-Δd ismeasured and compared to the threshold. For example, for a givendirection, this can be described as:|Z _(i)−2Z _(i+1) +Z _(i+2) |<st  (3),where Z represents a depth value, i represents a sample index along thedirection, s represents a scale factor, and t represents the samethreshold as in Equation (1).

If the difference (absolute value) is greater than or equal to s t, thenthe given depth sample is not connected to the contiguous depth sample,and if it is less than s t, then the given depth sample is connected tothe contiguous depth sample via a continuous underlying surface.

Once the connections between the depth samples are determined, a flag isset for each connection. Each depth sample may have up to 8 flags. It isunderstood that the threshold value may be predefined, e.g. set beforethe method 500 starts, or set dynamically, e.g. during the method 500.

From this point, the method 500 can be broken down into three passes.For Pass 1, the method 500 computes color values for unmapped depthsamples that are surrounded by diagonally contiguous color samples. ForPass 2, the method 500 computes color values for unmapped depth samplesthat are surrounded by horizontally/vertically contiguous color samples.For Pass 3, the method 500 assembles the computed color values for theunmapped depth samples with color values for the original color samples(originally rendered color samples) and selectively (based on theirconnections) sharpens/modifies them.

At step 530, for each unmapped depth sample surrounded by fourdiagonally contiguous color samples, e.g. 570-X, its connection(s) withthose four diagonally contiguous color samples are determined using theconnection information, e.g. the flags for the unmapped depth samples,from the step 520. The connections between the unmapped depth sample andthe diagonally contiguous color samples may be determined by checkingthe flags of the unmapped depth sample with respect to the contiguousdepth samples that have been mapped to respective diagonally contiguouscolor samples. FIG. 5E illustrates how an unmapped depth sample, e.g.,570-X, may be connected to four diagonally contiguous color samples,e.g., 562-a, 562-b, 562-c, and 562-d. In one embodiment, theconnection(s) between the unmapped depth sample and the diagonallycontiguous color samples may be in the form of a direction continuitymask that indicates whether a surface underlying the samples iscontinuous.

Once the connections between the unmapped depths samples and thediagonally contiguous color samples are determined, an interpolationmethod for calculating a color value for each of the unmapped depthsamples is selected at step 540. In the illustrated embodiment, theinterpolation method is selected based on a number of the connectionsbetween the unmapped depth sample, e.g., 570-X, and its diagonallycontiguous color samples, e.g., 562-a, b, c, d, which is determined atthe step 530. A number of the connections between each unmapped depthsample and its diagonally contiguous color samples can vary from zero tofour.

When the number of the connections is zero or four, the unmapped depthsample is treated as an interior sample and any reasonable (even linear)image scaling algorithm may be used. One such algorithm is exemplarilyillustrated in FIGS. 5F-N. The illustrated algorithm is an “offset”image scaling algorithm that for 4× scaling, i.e., 2 in each dimension,it only generates three new color values and reuses the existing fourthcolor. Normally, all of the pixels values are replaced with new ones ina linear algorithm.

The illustrated algorithm uses a pair of image operations, a non-lineardirectional interpolation D(Image, m) followed by an optional linearsharpening S(Image, s) to scale an image by a power of two in eachdimension. m and s are the two parameters used by the algorithm. Theillustrated algorithm's performance is typically in the gigapixel persecond range and offers quality visibly superior to purely linearscaling. It is noted that the illustrated method is performed in linearcolor space.

FIG. 5F illustrates a grid 578 with interior unmapped depth samples 575,576, and 577 and the color samples, i.e., shaded squares. For Pass 1,the algorithm will compute color values of unmapped depth samples thatare surrounded by diagonally contiguous color samples, e.g., 575. ForPass 2, the algorithm will compute color values of unmapped depthsamples surrounded by horizontally and vertically contiguous colorsamples, e.g., 576 and 577, by logically rotating the grid 578 45°counter-clockwise. The grid 578 that has been logically rotated forvertically contiguous color samples, e.g., 576, is shown in FIG. 5G. Thegrid 578 that has been logically rotated for horizontally contiguouscolor samples, e.g., 577, is shown in FIG. 5H. It is noted that any ofthe standard texture “out of range” operations, e.g., clamp, mirror,warp, border, etc., may be used to obtain data from outside theoriginal, i.e., rendered image. A clockwise rotation can be choseninstead.

For each interior unmapped depth sample, e.g. a circled depth sample575, two metrics, which are the sum of seven absolute differences of theluminances of the indicated diagonally-adjacent mapped color samples,are computed. The first metric, i.e., the sum of absolute differences ofthe luminances between seven 45° pairs of the mapped color samples582-1, 2, 3, 4, 5, 6, 7 and their diagonally-adjacent immediateneighbors in FIG. 5I is calculated as:

$M_{45} = {\sum\limits_{{45{^\circ}\mspace{14mu}{pairs}\mspace{14mu} i},j}{{{lum}_{i} - {lum}_{j}}}}$where for each 45° pair, i represents a color sample and j represents anadjacent color sample at 45° from the color sample at i. One such pairis shown in FIG. 5I as the “i sample” and “j sample.” The second metric,i.e. the sum of absolute differences of the luminances between seven135° pairs 584-1, 2, 3, 4, 5, 6, 7 in FIG. 5J is calculated as:

$M_{135} = {\sum\limits_{{135{^\circ}\mspace{14mu}{pair}\; s\mspace{11mu} i},j}{{{lum}_{i} - {lum}_{j}}}}$where for each 135° pair, i represents a color sample and j representsan adjacent color sample at 135° from the color sample at i.

It is noted that the metric is identical to convolving the renderedimage with kernels abs⊗

$\quad( \begin{bmatrix}1 & 0 \\0 & {- 1}\end{bmatrix} )$and abs⊗

$\quad( \begin{bmatrix}0 & 1 \\{- 1} & 0\end{bmatrix} )$and then applying an averaging filter e.g,

$\quad{\begin{matrix}0 & 1 & 1 \\1 & 1 & 1 \\1 & 1 & 0\end{matrix}}$to the result. The averaging filter is an improvement over a 3×3 boxfilter. If the metric is “small”, the image is isophotic in thatdirection. A small metric means that the sum of these absolute valuesubtractions is small, which means that each pair of values used wasclose to each other. “Isophotic” means “equally bright or illuminated.”

The interpolation is performed in the direction that has the smallestchange in brightness, e.g., not across an edge. The interpolation isperformed in the direction of the dominant isophote. If no direction isdominant, a standard linear filter is used for interpolation. Adominating direction is considered by comparing relative magnitudes.Thus, given the two metrics, a graph may be classified as shown in FIG.5K, where F₄₅ represents when 45° is the dominant direction, F₁₃₅represents when 135° is the dominant direction, and F_(4×4) representswhen there is no dominant direction. As such, if M₁₃₅>m M₄₅, F₄₅interpolation along 45° is performed as shown in FIG. 5L, if M₄₅>m M₁₃₅,F₁₃₅ interpolation along 135° is performed as shown in FIG. 5M, andF_(4×4) interpolation is performed as shown in FIG. 5N. The parameter mis greater than or equal to 1.0, and is typically chosen to be in therange of 1.1 to 1.5. m is another algorithm parameter.

In the illustrated embodiment,

$\lbrack {{- \frac{1}{16}},\frac{9}{16},\frac{9}{16},{- \frac{1}{16}}} \rbrack$are used as the weights for the F₄₅ and F₁₃₅ interpolations. For F_(4×4)interpolation, these same weights are used for the separable horizontaland vertical weights. It is noted that the weights are not limited tothe above weights and other weights in the approximate range

$\lbrack {{- \frac{1}{8}},\frac{5}{8},\frac{5}{8},{- \frac{1}{8}}} \rbrack$to

$\lbrack {{- \frac{1}{32}},\frac{17}{32},\frac{17}{32},{- \frac{1}{32}}} \rbrack$may also be used.

Once directionally interpolated, the sharpen filterp′=p+s(p−blur_(N×N)(p)) is applied to all channels of the pixel p thatcorresponds to the unmapped depth sample 575. This sharpen filter isapplied only if the unmapped depth sample 575 is connected to all of itscontiguous neighbors. The result is clamped to the representable rangeof p. In the illustrated embodiment, s is greater than or equal to 0.0,and is typically around 2.0. The blur filter is applied to a N×Nneighborhood centered at p; N is odd. A blur filter

${\frac{1}{16}\quad}{\begin{matrix}1 & 2 & 1 \\2 & 4 & 2 \\1 & 2 & 1\end{matrix}}$is a reasonable one to use (N is 3 in that case) but any filter thatqualifies as a low-pass filter may be used. s is another algorithmparameter.

In another embodiment, an interpolation method using a kernel or aGaussian filter can be selected for an interior sample. To use an imagekernel, error terms for kernels that interpolate along variousdirections, e.g. horizontal/0 degree, 22.5 degree, 45 degree, 67.5degree and vertical/90 degree, are first calculated. An error term foreach kernel represents a color variance in each respective direction andis calculated as a sum of the absolutes of the difference in color valuealong each respective direction. The kernel with the minimal error termis selected to calculate the color value of the unmapped interior depthsample. The Gaussian filter may be used when there is no dominantdirection.

When the unmapped depth sample is connected to at least one, but notall, of the diagonally contiguous color samples, the unmapped depthsample is a non-interior, e.g. an edge sample, and an interpolationmethod using the connected diagonally contiguous color sample(s) isselected.

When the unmapped depth sample is connected to one diagonally contiguouscolor sample, an interpolation method using the color value of that onediagonally contiguous the color samples is selected. When the unmappedsample is connected to two contiguous color samples, an interpolationmethod using the mean color value of those two diagonally connectedcolor samples is selected. When the unmapped depth sample is connectedto three diagonally contiguous color samples, an interpolation methodusing the mean value of two of the three diagonally contiguous colorsamples that are located on a long diagonal is selected. For example,referring back to FIG. 5E, if the unmapped depth sample 570-X isconnected to three contiguous color samples 562-b, c, d, aninterpolation method that uses the mean value of the color samples 562-band c, which are on the long diagonal, is selected. Using two samplesinstead of three fixes a significant number of artifacts. At step 550,color values for each unmapped depth samples are calculated using theselected interpolation method of the step 540.

As the steps 530-550 are performed for unmapped diagonal depth samplessuch as 570-X, which are surrounded by four diagonally mapped colorsamples, they correspond to Pass 1. Once color values for the diagonaldepth samples are determined, the steps 530-550 are repeated in Pass 2for unmapped horizontal and vertical depth samples, which are surroundedby four diagonally/horizontally mapped color samples, e.g. 570-Ch and570-Cv in FIG. 5C. For Pass 2, the input data would be rotated 45degrees counterclockwise from Pass 1. As such, instead of using theconnections with the diagonally contiguous color samples as in Pass 1,Pass 2 would use the connections with horizontally and verticallycontiguous color samples. For example, to determine the color value ofthe unmapped horizontal depth sample 570-Ch located directly below570-X, Pass 2 would use the connections with the horizontally andvertically contiguous color samples, e.g. the calculated color samplecorresponding to 570-A (located directly below the aforementioned570-Ch), 570-X, and the color samples 562-c and d. The steps 530-550 maybe performed in a similar fashion when the input data is rotated 45degrees clockwise.

Once the color values of all the unmapped depth samples are calculated,they are assembled with the color values from the original/renderedcolor samples and selectively modified, e.g. sharpened. It is noted thatnot all the color values are modified because modifying all colorsample, especially those of edge samples, may create halo artifacts. Assuch, using the connection information from the step 520, only colorvalues of color samples that are connected to all of their neighbors aresharpened. The step 550 corresponds to Pass 3.

The method 500 ends at step 555. The color values of unmapped depthsamples determined from the method 500, which represent color values forthe upscaled image, may be anti-aliased or displayed as described abovewith respect to FIG. 4.

The aforementioned method 500 describes the 4× upscaling. The method 500may also be used for 2× upscaling with a slight modification. Similar tothe 4× upscaling, the 2× upscaling also uses color samples that aregenerated at a first resolution and depth samples that are generated ata second resolution. The first resolution of the color samples would beat a half resolution of the second resolution to achieve a 2:1 mappingbetween the depth and color samples, whereas in the 4× upscaling, thefirst resolution was at a quarter resolution of the second resolutionfor a 4:1 mapping. To provide the color samples at a half resolution ofthe depth samples, the color samples are generated twice at a quarterresolution of the second resolution, first for the Original positions,i.e., locations of unshaded circles 590 in FIG. 5O, and second for theDiagonal positions, i.e., locations of shaded circles 592 in FIG. 5O.When the color and depth samples are mapped in a single grid, the grid595 appears as a checker board as illustrated in FIG. 5O. Circlesrepresent color samples, squares represent depth samples, andcircled-squares represent mapped color and depth pairs.

It is noted that since the color values at the shaded circles 592corresponds to the color values determined at Pass 1 of the method 500,the grid 595 in FIG. 5O can be considered as the same grid in FIG. 5Bafter Pass 1. As such, 2× upscaling with the 2:1 mapping may beimplemented by using only Pass 2 and 3 of the method 500.

The filter map for the 2× implementation, i.e. 2:1 depth buffer to colorbuffer ratio, can utilize a variety of algorithms. Two example methodsare described below. In an embodiment where a guided image filtering isused to sharpen the image, then the algorithm can send 49 cases (7 casesfor each of the horizontal and vertical outputs) which results in atotal of 6 bits per 2×2 tile. The seven cases are shown in FIG. 10, notincluding the case, which is not needed (since the client determines ifsharpening is needed or not for each color sample.) In an embodimentwhere the sharpening bit is determined on the server, then 2 bits (onebit for each input color sample indicating whether to sharpen it or not)and 6 bits (8 cases stored in three bits for each output color sample)is sent for a total of 8 bits per 2×2 tile. Method 500 and FIG. 5B to 5Jprovide examples of the different interpolation cases that can berepresented in a filter map based on connectivity information. FIG. 10illustrates the 8 different cases for a filter map and the additionalinformation of sharpening when determined on the server. In thisembodiment, the rightmost two cases (and interpolator) indicate,respectively, that the color sample is not to be sharpened, or is to besharpened.

FIG. 6 illustrates a flow diagram of an example method 600 for managingserver system resources with respect to rendering a video. Method 600starts at a step 601 and proceeds to a step 605 where the system willdetermine various user parameters for the client computing device, forexample, the client video capability, including the ability to performimage corrections, the client computing device bandwidth andconnectivity parameters, the client computing device display parameters,the targeted minimum FPS parameters, and the client computing device'stier level of service utilizing the user parameters. These factors canbe utilized by a decision step 615.

Proceeding to a step 610, the method 600 determines a time when videorendering processes should be shifted from the server system to theclient computing device. This determination can utilize factors such asthe maximum number of VMs a server system can support before there issystem response degradation, the type of applications running,processing demand for server system resources, such as memory, CPUcycles, GPU cycles, communication bandwidth, and other factors. Decisionstep 615 utilizes the data and parameters collected in steps 605 and 610to determine when a video should be rendered on the server system or onthe client computing device.

If decision step 615 resultant is a ‘Yes’, the method 600 proceeds tosteps 630 and 650, which can be executed sequentially, overlapping, orin parallel. Step 630 pre-processes rendering data, such that therendering data is processed through a selected compression algorithm.For example, the rendering data processing can include an algorithm togenerate a color buffer, a depth buffer, and a filter or pixelconnectivity map. In other embodiments, other algorithms can be applied,for example, single bit encoding for edge sensitive sharpening,compressing tile blocks, creating arrays of bitmaps, applying losslessbit-plane compressions, and other algorithms now known or laterdeveloped. The rendering data is what will be used at the clientcomputing device to generate images to display on the client computingdevice. For example, the rendering data can be a color buffer andconnectivity information regarding pixels of the color buffer.

In a step 632, the processed rendering data is sent to the clientcomputing device. Proceeding to a step 634, the client video componentprocesses the rendering data by executing a selected algorithm. Suchprocessing of the rendering data can include decompressing the data,decoding the data, rendering the video, and upscaling the video. In astep 636, the client video component can apply various algorithms andtechniques to correct the video image, such as correcting compressionartifacts, applying spatial and temporal anti-aliasing, and modifyingthe algorithm parameters. In a step 640, the final video is sent to theclient display.

In the step 650, the server system, can allocate system resources, asnecessary, to achieve desired operational goals on the server system.For example, a GPU can be allocated to a different VM to maintainoverall performance throughput of the running applications. In otherembodiments, additional VMs can be created and supported with theresources available on the server system.

Returning to the decision step 615, if the resultant is ‘No’, then themethod 600 proceeds to a step 620 where the rendering data is renderedon the server system. In a step 625, the video is compressed usingstandard techniques and is sent to the client computing device fordisplay. In this path, the client video component un-compresses thevideo stream and utilizes algorithms for upscaling and correcting thevideo image.

Proceeding to the step 640, the client computing device displays thevideo. The method 600 ends at a step 660.

FIG. 7 illustrates a flow diagram of an example method 700 for the usageof various compression bit stream algorithms. In this example, method700 is inserted between steps 630 and 632 of method 600. The method 700begins at a step 701 which indicates that method 700 is part of another,for example, method 600. Step 630 is executed as described previously.Proceeding to a step 710, the video processor, as represented in diagram800 as video processor 830, determines the appropriate compressionbit-stream algorithm to be utilized. The determination can utilizevarious factors and parameters, such as the capability of the serversystem video processor, the resources available on the server system,the client computing device capability, and other parameters.

Method 700 describes five example algorithms, though additionalalgorithms can be utilized. Each of the described compression bit-streamalgorithms include a type of implementation for a low resolution colorbuffer and higher resolution depth buffers, each of which is determinedand calculated at the server system.

Path 1: Proceeding to a step 720, a connectivity bitmap can be created.This bitmap can indicate the connections between color pixels within thecolor buffer. The bitmap can be partitioned into 2×2 tiles of colorpixels. The partitioned bitmap can then be encoded, for example, byusing twelve bits with four additional bits indicating whether the pixelis an edge pixel. This can result in a representation of sixteen bitsper 2×2 tile. This is a reduction from the thirty-two bits required toencode an uncompressed 2×2 tile (i.e. utilizing one bit for each filtermap connection direction for each pixel in the 2×2 tile). Proceeding toa step 722, the 2×2 tiles determined in the step 720 (or a step 730, asappropriate) are collected into blocks of eight by eight (8×8) of the2×2 tiles. If the 8×8 block does not contain an edge pixel, then theentire block can be denoted using one bit. Otherwise, the block data isused, without further compression, to denote the pixels. Proceeding to astep 724, a compression algorithm, for example, ZIP, or othercompression algorithms, can be applied to the result of the step 722.The method 700 proceeds to a step 632.

Path 2: Proceeding to a step 730, the anchor, i.e. original, pixel fromeach 2×2 color tile can be represented by a single bit indicatingwhether that 2×2 tile should have a sharpening algorithm applied. The2×2 tile can then be encoded utilizing a filter or pixel connectivitymap for the remaining 3 pixels. The filter or pixel connectivity mapresults in twelve different values that can be encoded: one value is forapplying a compression algorithm and request sharpening, one value isfor applying a compression algorithm with no sharpening, four values forusing one of four adjacent pixels without sharpening, and six valueswhere two of four adjacent pixels are used without sharpening. This canreduce the required number of bits for encoding the 2×2 tile to twelvebits. The method 700 then proceeds to the step 722. In anotherembodiment, four of the twelve values can be removed as not affectingthe visual output to a degree noticeable by a user. Therefore, only 8values need to be encoded. This results in a total of 10 bits per 2×2tile.

Path 3: After executing the step 730, the method 700 proceeds to a step742. In the step 742, the 2×2 tiles determined in the step 730 arecollected into blocks of 8×8 of the 2×2 tiles. Step 742 then executesrecursively, i.e. a quadtree encoder is executed. Step 742 firstdetermines one coded bit for each 8×8 block. Then the 8×8 block is splitinto four, four by four (4×4), blocks, each with its own coded bit. Each4×4 block is then split into 2×2 tiles. At this point, each non-zerovalue can be variable length coded using an algorithm, for example,Huffman encoding. The coded bits are then combined. The method 700 thenproceeds to the step 632.

Path 4: Proceeding to a step 750, since sharpening occurs for fullyconnected pixels, the initial encoding can be completed using arrays,i.e. bitmaps. The first array is determined by a single bit per pixelbitmap, where, for example, a bit value of zero represents a ‘do notsharpen’ state and a bit value of one means a ‘sharpen’ state.Proceeding to a step 752, a second array can be determined using values,for example, zero to ten, per pixel. The second array requires data forthose corresponding bits in the first array that have a specified value,for example, zero. Proceeding to a step 754, a lossless bit planecompression technique can be applied. The resulting first and secondarrays (bitmaps) can have sequences, areas, blobs, and curves containingcontiguous zeros or ones. This inherent bitmap structure permitscompression algorithms to perform better. The first and second arraysare then compressed, similar to the step 724. The method 700 thenproceeds to the step 632.

Path 5: Proceeding to a step 760, a connectivity bitmap can be created.This bitmap can indicate the connections between color pixels within thecolor buffer. The bitmap can be partitioned into tiles of two by two(2×2) color pixels. The 2×2 tile can have an original pixel and threeremaining pixels. Proceeding to a step 762, a color for the threeremaining pixels can be determined, through using a calculated value oftwo color pixels or applying an interpolator algorithm.

There are additional compression bit stream algorithms that can beutilized. Not shown is an algorithm similar to that of path 4. For thesecond array, i.e. step 752, values 0-11 are stored for three of thedepth values with one extra bit for the fourth depth value. This type ofarray can fit into twelve bits per 2×2 tile and can also be stored intothirteen bits, in a pattern of (4,4,4,1) which can make subsequentcalculations easier and faster. The remaining steps of path 4 remain asdescribed therein.

Also not shown is an algorithm similar to that of path 4. For the secondarray, i.e. step 752, values 0-7 are stored for three of the depthvalues. If there is only one color, then the process can useinterpolation. If there are more than one color then the calculatedvalue of two pixels is evaluated, or, in an alternative embodiment, aninterpolation algorithm can be applied. A sharpening algorithm can beapplied or can be skipped. An additional bit is used for the fourthdepth value. This type of array can fit into ten bits for each 2×2 tile.This algorithm can also be utilized with a 2× depth ratio therebyutilizing eight bits per 2×2 tile.

Also not shown is an algorithm similar to that of path 4. For the secondarray, i.e. step 752, values 0-6 are stored for three of the depthvalues. An interpolator is applied. A sharpening algorithm can then beapplied that ignores depth values, for example, an edge-sensitivesharpening algorithm, guided image filtering, or edge-threshold unsharpmasking. This embodiment uses 9 bits per 2×2 tile.

In all embodiments, the bitmapping process can reduce the number of bitsrequired to represent the filter map by increasing the threshold t, i.e.the threshold used when computing the filter or pixel connectivity map.In other words, as more pixels are identified as being connected, i.e.fewer edge pixels identified, fewer bits are needed to represent thosepixel groups.

Proceeding to a step 764, an edge sensitive sharpening algorithm can beapplied. Then a compression and encoding algorithm can be applied to thefinal result. The method 700 then proceeds to the step 632.

Proceeding to the step 632, the method 700 continues processing orexecuting, for example, the method 600. The method 700 ends as themethod 600, or other method, continues its operations, as represented bystep 790.

FIG. 8 illustrates a block diagram of an example video rendering system800 capable of delivering a video to a client computing device. Thevideo rendering system 800 includes a server system or set of serversystems 810 and at least one client computing device 840. Server system810 includes a processor 815, an allocator 816, a verifier 817, avirtual machine group 820, virtual machine instances 825, server systemresources 827, a video processor 830, and a video pre-processor 835.Components 815, 816, 817, 820, 825, 830, and 835 represent a logicalseparation of functionality and these components can exist physically invarious combinations, together, adjacent, or separated. In addition,these components can exist on one or more separate server systems thatare communicatively coupled.

Client computing device 840 includes a video processor 842, a videorenderer 844, and a client display 845. Client computing device 840 canbe communicatively coupled to server system 810 by a communicationsconnection 850. Communication connection 850 can be various availablecommunication types, such as through the internet, wide area network,private network, a direct cable connection, such as Ethernet, andthrough a wireless connection.

Server system 810 can be located in one or more physical locations, forexample, a data center, a cloud environment, proximate to the clientcomputing device, and other locations. Client computing device can belocated proximate to or separate from the server system 810. Forexample, the client computing device can be located in a user's houseand the server system located in a cloud environment, or the serversystem and client computing device can be located proximate each othersuch as in a conference center.

An application or applications can be actively running in one or more ofthe VMs 825, of which there can be various numbers of VMs as part of825, within the virtual machine group 820. Each of the VMs 825 requirethe use of some server system resources 827. For example, the resources827 can be memory of various types, such as cache or general memory, CPUcores and cycles, GPU cores and cycles, network bandwidth, server systempipeline access, database access, permanent or magnetic based storage,such as hard drives, and other resources to support the running andoperation of the VMs. The server system resources 827 can be limited asto their availability at various points in time in the operation of theVMs 825. The allocation of the server system resources 827 among the VMs825 is the responsibility of the allocator 816. As part of allocator816's determinations, it can, working in conjunction with processor 815,guide one or more VMs 825 to shift some video rendering tasks to theclient computing device 840 thereby increasing the availability ofserver system resources 827 to the VMs 825 and allowing the potentialcreation of additional VMs 825.

Processor 815, which can be one or more processors located together orseparately, provides for the control of the virtual machine group 820and executes the methods described herein. As an application executesand a video is to be displayed for a user of the client computing device840, the processor 815 determines, utilizing resource allocator 816,computational load, client computing system 840 capability, user tierlevel of service, and other factors, where the rendering data should berendered to provide the desired FPS throughput for the user while notover allocating the server system resources 827.

In an alternative, video processor 830 can utilize pre-processor 835 topre-process the rendering data by applying compression bit-streamalgorithms and techniques to deliver a compressed rendering data toclient computing device 840. Video processor 842, working with renderer844, can decode, decompress, upscale, apply image corrections due tocompression artifacts, to the rendering data, and otherwise prepare thevideo for display on the display 845. This alternative path shifts aportion of the video rendering to the client computing device 840 fromthe server system 810.

Verifier 817 is a logical process that allows the application VM torequest information from the client computing system 840 to identifyparameters to be used by the processor 815 and video processor 830. Forexample, verifier 817 can determine the available bandwidth provided bycommunication connection 850, can determine the display 845 resolution,and can determine the capabilities and protocols supported by videoprocessor 842. These parameters can be used to select an appropriatecompression algorithm and to determine the parameters to be utilizedwith the selected compression algorithm.

FIG. 9 illustrates a block diagram of an example compression algorithmsystem flow 900. Flow 900 builds on the system rendering system 800.Video processor 830 can execute various algorithms and one suchalgorithm is demonstrated in flow 900. A low resolution color buffer 910can be determined and then passed to a video encoder 912. A highresolution depth buffer 915 can be determined and then the depth buffercan be partitioned to create a filter map 917. The result can then bepassed to a lossless encoder.

The output from the lossy video encoder 912 and lossless encoder 919 canbe passed, through communications connection 850, to computing device840 and more specifically to video processor 842.

Video encoder 912's output can be passed to video decoder 930, which inturn passes its output to the lossy low resolution color buffer 932.Lossless encoder 919's output can be passed to the filter map decoder935, which in turn passes its output to the filter map 937. Buffer 932and map 937 each pass their respective outputs to renderer 844 which canexecute the rendering algorithm 940. The output of algorithm 940generates a high resolution color buffer 942. Buffer 942's output isthen further processed by video processor 842 and prepared for displayon display 845 (shown by the unconnected arrow).

In interpreting the disclosure, all terms should be interpreted in thebroadest possible manner consistent with the context. In particular, theterms “comprises” and “comprising” should be interpreted as referring toelements, components, or steps in a non-exclusive manner, indicatingthat the referenced elements, components, or steps may be present, orutilized, or combined with other elements, components, or steps that arenot expressly referenced.

Those skilled in the art to which this application relates willappreciate that other and further additions, deletions, substitutionsand modifications may be made to the described embodiments. It is alsoto be understood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting, since the scope of the present disclosure will be limited onlyby the claims. Unless defined otherwise, all technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which this disclosure belongs. Although anymethods and materials similar or equivalent to those described hereincan also be used in the practice or testing of the present disclosure, alimited number of the exemplary methods and materials are describedherein.

It is noted that as used herein and in the appended claims, the singularforms “a”, “an”, and “the” include plural referents unless the contextclearly dictates otherwise.

The above-described apparatuses, systems or methods or at least aportion thereof may be embodied in or performed by various processors,such as digital data processors or computers, wherein the processors areprogrammed or store executable programs or sequences of softwareinstructions to perform one or more of the steps of the methods orfunctions of the apparatuses or systems. The software instructions ofsuch programs may represent algorithms and be encoded inmachine-executable form on non-transitory digital data storage media,e.g., magnetic or optical disks, random-access memory (RAM), magnetichard disks, flash memories, and/or read-only memory (ROM), to enablevarious types of digital data processors or computers to perform one,multiple or all of the steps of one or more of the above-describedmethods or functions of the system described herein.

Certain embodiments disclosed herein or features thereof may furtherrelate to computer storage products with a non-transitorycomputer-readable medium that has program code thereon for performingvarious computer-implemented operations that embody at least part of theapparatuses, the systems, or to carry out or direct at least some of thesteps of the methods set forth herein. Non-transitory medium used hereinrefers to all computer-readable media except for transitory, propagatingsignals. Examples of non-transitory computer-readable medium include,but are not limited to: magnetic media such as hard disks, floppy disks,and magnetic tape; optical media such as CD-ROM disks; magneto-opticalmedia such as floptical disks; and hardware devices that are speciallyconfigured to store and execute program code, such as ROM and RAMdevices. Examples of program code include both machine code, such asproduced by a compiler, and files containing higher level code that maybe executed by the computer using an interpreter.

What is claimed is:
 1. A method of upscaling an image, comprising:rendering an image, wherein said rendering includes generating colorsamples of said image at a first resolution and depth samples of saidimage at a second resolution, which is higher than said firstresolution; and upscaling said image to an upscaled image at a thirdresolution, which is higher than said first resolution, using said colorsamples and said depth samples, wherein said upscaling includescomputing connections between said depth samples by determining firstslopes between said depth samples and contiguous depth samples anddetermining second slopes between said contiguous depth samples anddirectionally adjacent depth samples.
 2. The method of claim 1 furthercomprising: displaying said upscaled image at a native resolution of adisplay device, wherein said third resolution is said native resolutionof said display device.
 3. The method of claim 1, wherein said thirdresolution is said second resolution.
 4. The method of claim 1, whereinsaid upscaling further includes: mapping said color samples to saiddepth samples; determining color values for unmapped depth samples ofsaid depth samples based on said mapping and said connections; andassembling color values for mapped depth samples of said depth sampleswith said color values for said unmapped depth samples.
 5. The method ofclaim 4, wherein computing said connections between said depth samplesincludes: calculating differences between said first slopes and saidsecond slopes; and comparing absolute values of said differences to ascaled threshold value.
 6. The method of claim 4, wherein when saidunmapped depth samples are interior depth samples, said determining saidcolor values includes: calculating error terms for multiple kernels thatinterpolate along multiple directions; selecting one of said multiplekernels that has a minimum error term; and convolving said one kernelalong a respective direction.
 7. The method of claim 1, wherein saidgenerating said color samples includes generating said color samplestwice at said first resolution.
 8. The method of claim 1, wherein saidgenerating said color samples includes: selecting a fragment thataffects a color value of one of said color samples based on at least oneof attributes of said fragment; and writing to said color value of saidone color sample using said fragment.
 9. The method of claim 8, whereinsaid attributes include a pre-specified sample ID of said fragment, anAPI order of said fragment, a depth value of said fragment, a coveragemask of said fragment, and a size of said fragment.
 10. The method ofclaim 1, further comprising anti-aliasing said upscaled image, andwherein said anti-aliasing is carried out in a single pass with saidupscaling.
 11. The method of claim 10, wherein said anti-aliasingincludes: dividing color samples of said upscaled image into groups; anddetermining an average color value for each of said groups.
 12. Themethod of claim 10, wherein said anti-aliasing includes obtaining ananti-aliased image by applying an anti-aliasing filter to color samplesof said upscaled image, wherein said anti-aliased image is at aresolution lower than said third resolution.
 13. The method of claim 1,wherein said color samples and said depth samples correspond to oneanother via locations of said color samples and said depth samples in agrid.
 14. A system for upscaling an image, comprising: a processor thatincludes: a rendering engine configured to render an image by generatingcolor samples of said image at a first resolution and depth samples ofsaid image at a second resolution, which is higher than said firstresolution; and an upscaling engine configured to upscale said image toan upscaled image at a third resolution, which is higher than said firstresolution, using said color samples and said depth samples, whereinsaid using includes computing connections between said depth samples bydetermining first slopes between said depth samples and contiguous depthsamples and determining second slopes between said contiguous depthsamples and directionally adjacent depth samples; and a memory coupledto said processor and that includes a color buffer storing said colorsamples and a depth buffer storing said depth samples.
 15. The system ofclaim 14 further comprising a display device configured to display saidupscaled image at a native resolution of said display device, whereinsaid third resolution is said native resolution of said display device.16. The system of claim 14, wherein said third resolution is said secondresolution.
 17. The system of claim 14, wherein said using furtherincludes: mapping said color samples to said depth samples; determiningcolor values for unmapped depth samples of said depth samples based onsaid mapping and said connections; and assembling color values formapped depth samples of said depth samples with said color values forsaid unmapped depth samples.
 18. The system of claim 17, wherein saidconnections between said depth samples are computed by: calculatingdifferences between said first slopes and said second slopes; andcomparing absolute values of said differences to a scaled thresholdvalue.
 19. The system of claim 17, wherein when said unmapped depthsamples are interior depth samples, said color values is determined by:calculating error terms for multiple kernels that interpolate alongmultiple directions; selecting one of said multiple kernels that has aminimum error term; and convolving said one kernel along a respectivedirection.
 20. The system of claim 14, wherein said color samples aregenerated twice at said first resolution.
 21. The system of claim 14,wherein said color samples are generated by selecting a fragment thataffects a color value of one of said color samples based on at least oneof attributes of said fragment and writing to said color value of saidone color sample using said fragment.
 22. The system of claim 21,wherein said attributes include a pre-specified sample ID of saidfragment, an API order of said fragment, a depth value of said fragment,a coverage mask of said fragment, and a size of said fragment.
 23. Thesystem of claim 14, wherein said processor further comprises ananti-aliasing engine that is configured to divide color samples of saidupscaled image into groups and determine an average color value for eachof said groups.
 24. The system of claim 23, wherein said anti-aliasingengine is further configured to apply an anti-aliasing filter to colorsamples of said upscaled image to obtain an anti-aliased image, whereinsaid anti-aliased image is at a resolution lower than said thirdresolution.
 25. The system of claim 14 wherein said color samples andsaid depth samples correspond to one another via locations of said colorsamples and said depth samples in a grid.
 26. A non-transitorycomputer-readable medium storing executable instructions that, whenexecuted by one or more processors, cause said one or more processorsto: render an image by generating color samples of said image at a firstresolution and depth samples of said image at a second resolution, whichis higher than said first resolution; and upscale said image to anupscaled image at said second resolution using said color samples andsaid depth samples, wherein said using includes computing connectionsbetween said depth samples by determining first slopes between saiddepth samples and contiguous depth samples and determining second slopesbetween said contiguous depth samples and directionally adjacent depthsamples.